Yes, dream mapping seems one approach that shamans have used in the former Soviet republic, across Eastern Europe to Siberia.  I first heard about this from a  Ukrainian who came to an International Association for the Study of Dreams (IASD) conference.  I mention the person and his presentation in Chapter 6 of Lucid Dreaming – Plain and Simple (and the endnote about the presentation is #65).
He explained that in the shamanic tradition there, they felt there was an Upper World, a Middle World and a Lower World (each with its own characteristics).  He said the shamans felt that a person could use ‘dream mapping’ to explore these worlds successfully, and learn secrets, gain power, deal with issues, etc.
The person explained that he got hexagonal graph paper (small hexagons, connect all across it) — and in the center was HOME.  Then in each of his dreams, he marked where they occurred, relative to (his felt sense of the direction to) HOME.  After a few months, he began to have an outline of what happened in all the directions, and precisely where.
Then he said, he began to get lucid, by realizing, “Have I seen this space before?” — and remembering that he had in a previous dream, which reminds him that he must be dreaming now.
Also, that gave him the capacity to ‘remember’ portals that seemed to exist to the other worlds (Upper and Lower) — and so he could find them, and move dimensionally.
I believe if you do some considerable research, you will find some anthropologists who can flesh out the idea of Worlds — and on rare occasion, those who can discuss it in terms of shamanic journeying and lucid dreaming.
One issue with some of this — certain teachings are only taught to initiates or tribal members.  So you may discover that you need assistance with firsthand sources, if you can find them.

When I have an online workshop and someone has an issue occur in a lucid dream, I normally ask them to describe “exactly” what happened (and their thinking during the lucid dream at that moment).  I ask this because the lucid dream ‘mentally reflects’ our beliefs, expectations, focus and intent — in that moment.
Also I ask because sometimes they already (inwardly) know that they feel conflicted about the question — or feel it is improper to get information in the dream state.
In general if lucid dreams are normally mentally reflective, then a person needs to examine what happens in response to their query.  For example, do you not want to hear the response?  Does your ego know what it wants to do, and therefore fights an alternate view?  All of these things can lead to muffled responses, or the inability to speak, etc.  This means that the problem involves our inner conflict (sometimes, between wanting to know but not wanting to hear — or feeling concerned about the response).
For those lucid dreamers who reach out to the larger awareness with questions, then you may see that a certain category of question gets a lovely response (e.g., open-ended questions, like, ‘Hey dream, show me something important for me to see!’), while another category of questions, (e.g., ego derived preferences, like ‘Hey dream, show me if I should go to the movies or stay home and ….’) result in very little response.  The questions can be the issue.
Hope that is a bit helpful.  When I have a detailed report, then the issue often becomes obvious.

Thanks for sharing your experience!  I want to assure you that you are very close.
But an adjustment seems necessary.
In dreams and lucid dreams (and even the dream of life), we see that the environment reflects the mind.  A person with many inner conflicts will have dreams which reflect conflicts. A person who focuses on X will have dreams which reflect X. Simply stated, the dream environment reflects the mind (and its beliefs, expectations, focus, intent, etc.).
In your case, it appears the ‘technique’ has become the ‘goal’.  😉
You have done a wonderful job of focusing on lucid dreaming ‘technique’ — and what gets reflected in your dreams (?) — a further refinement of ‘technique’!  If only you find some wood and paint, then (and only then) will you become lucid.
So here you have a few choices on how to approach this (all of which involve changing the mind).
If you wish, you can begin to focus more on ‘experiencing lucid awareness’ (than on the technique to try and get to lucidity).  How do you do that?  Use your focus.  Ignore the part about ‘how to get lucid’ — and just begin to focus on the aware part of lucidity (e.g., ‘Wow, look at this lucid dreamer and how he flew, or jumped, or talked or realized or changed or transformed!  Wow!’).  By focusing on the lucidity, you move your focus and your mental energy there.
Then since the dream environment ‘reflects the mind’ — you will naturally find yourself there (ie., where you focus), lucidly aware.
To do this, you can read our free online magazine, Lucid Dreaming Experience, at www.DreamingLucid.com   The final 15 pages include reader submitted lucid dreams.  Focus on the ‘lucid part’.  As you put your focus and energy there, then you will find yourself easily getting there.
Learning to use the mind helps us to Live Lucidly.  Your experience already shows a wonderful inner teacher helps you.  Lucid wishes!

If you look at some similar questions on this site, you will see that many people have the same experience.  They do well at lucid dreaming, but then have droughts — and do not have a lucid dream for a month or more.
While this seems distressing, I feel it normally reflects one of these issues:
a) Your life has become busier or more stressful, and your ‘focus’ shifts to daily concerns so much that dream recall and lucid dreaming suffer,
b) A person may do something new (like begin taking medications for a condition, start drinking/smoking, etc.) which has a negative effect
c) A person may have success with a technique, but as the ‘technique’ becomes routine and mundane, it fails to engage the person in a way that leads to lucid dreams (so they need something more ‘engaging’ or an approach that seems ‘engaging’),
d) And Sometimes — if a person looks back and reads the last lucid dream or two, they will see some element of ‘fear’ or some element of refusing to grow/change (either literally or symbolically in the lucid dream).  In this case, the person may subconsciously refuse to recall dreams or refuse to lucid dream, until they ‘resolve’ the fear or concern.
e) Miscellaneous (you watched a movie/show or read an article that scared you about the subconscious, etc. and since then, your recall has been affected — or you have No Goals to achieve in lucid dreaming, and therefore you have no direction ).
To your other question about having a lucid dreaming routinely, I mention that I achieved this at one time (having 30 lucid dreams a month) by “Developing a Lucid Mindset”.  I assume if you google my name and those words, then the approach will show up.  It is a very simple technique but requires consistent mindfulness each day.
Finally, I encourage those who read my books to see lucid dreaming as a Journey, a long path.  It takes time, it takes inner growth, it takes removing limiting beliefs, it takes mindful engagement and playful work.

Thanks for your note and description of your experience.
Here is my suggestion:
-Do not try to become lucid.  So, do not perform any of the lucid dream induction practices.
-Instead, before sleep, tell yourself, “Tonight I will clearly remember the most important dream of the night.”  Repeat it softly in your mind, and make it your intent.
-Write down your dream (or dreams).
Many of us can remember “the most important dream” and then see that our regular dreaming continues without any trouble.  For you, I feel it is important to return to that understanding, and see that you continue to have regular dreams.
Regarding your experience of trying to lucid dream (and then it appears suddenly having no dream recall), I assume that a connection exists.  Since I can only guess as to the reasons why, my first guess would be that you are both attracted to and frightened by the idea of lucid dreaming (which means that you have an inner conflict about it). Or my second guess, lucid dreaming seems like a powerful spiritual practice and this both attracts and concerns you (so again, you have a bit of an inner conflict). The result?  You don’t remember anything.
Does that sound a little bit correct?
If so, then you might want to have a conversation with your self and try to resolve the concern or fear.  For example, you could tell yourself, “Millions of people have a lucid dream each night this week.  It’s easy and natural.  There is nothing to be concerned about.”
Or you could read the research paper by Ursula Voss of Lucid Dreaming in German School Children — where she engages 694 German school children between the ages of 6 and 19 — and finds that about 20% of the eight year olds have already had a lucid dream.  It’s natural.  Of the entire group, about 51% has had at least one lucid dream.  For most, it simply happened spontaneously.  Again, that means it is natural and okay for you  to allow yourself to have a lucid dream. 😉

You bring up a point that I address in both of my books.
For the science of lucid dreaming (and the rational lucid dreamer), I have suggested the following:
You can see ‘Levels of Creativity’ operating within the lucid dream.  For example, a lucid dreamer will notice that when they turn the corner on a street, a brand new street scene exists.  The lucid dreamer did not have to request it, or make it happen; instead, an automatic unconscious function (working on certain principles) creates a street scene.  Similarly, when a lucid dreamer flies through a wall, suddenly a new vista appears on the other side (without any conscious thought activity by the lucid dreamer).  So at the first level of lucid dream Creativity, we see an automatic, unconscious function (which works on certain principles, as described in my books).
At the second level of lucid dream Creativity, you see the lucid dreamer using ‘experience creating’ principles (like belief, expectation, intent, etc.) to make an object or event appear or disappear.  For example, a lucid dreamer might say, “When I open this garage door, I will see a red Ferrari!”  And he opens the garage door, and there is a red Ferrari, with the top down, leather seats and other things in exquisite detail.
At the third level of Creativity, as I mention in my books, one needs to go beyond their own limitations.  Here, they ignore dream figures and the dream environment while they reach out to the ‘second psychic system’ or awareness behind the dream — and make a request, such as “Show me something important for me to see!”  Often at that point, the entire lucid dream scene will change, and they will be looking at something important.  Or perhaps, one object will appear.  In my books, I show how myself and others have called on this infinitely more creative ‘awareness’ and experienced its creativity in microseconds (and also how its responses meet Jung’s criteria for an inner ‘second psychic system’.
So as you point out, within the dream bubble exists an experiencer and the larger experience.  When lucid, the experiencer can see how the process works, how to influence it at the level of their own thinking, and then how to influence it by calling upon a ‘second psychic system’ or inner awareness.

Yes, a person can enter into a lucid dream by watching the colors swirl, maintaining their awareness and then eventually a dream-scene may appear.  Sometimes, it may feel like you have to ‘move’ into the dream ( as if you have to pull yourself through a door way into the dream).   Here, if you ‘focus’ on where you wish to be, then it seems easier to get there.  So if you see a garden, and focus on the fig tree — then you can move into the dream aware, more easily.
Another thing to do:  I have also found that if I completely silence my mind, then the colors will not swirl — and normally one color will shine all across the visual field.
Also if you spontaneously wake at night, then with a relaxed and peaceful mind as you go back to sleep, you can quietly think, “One, this is a dream.  Two, this is a dream.  Three, this is a dream….”  And continue.  You may find yourself saying, “Twenty-one this is a dream” — and realize that a dream scene appears around you and now you are lucidly aware!  This technique was introduced by Stephen LaBerge.
I have read a bit by the Sufi teacher hundreds of years ago, Ibn Al Arabi.  From his writings, it seems that he learned much from his lucid dreaming and spiritual journeys.  I have also taught with the head of the Sufi Association in the United Kingdom (Nigel H.).
Yes, dreaming and lucid dreaming can be seen as a specific inner realm (with its own rules and principles) — and then there are other realms too, which may seem quite similar but are different (and therefore have their own rules and principles).  I remember once, I was ‘aware’ and being taught by a wise man, and I said to the man, “This is an incredibly stable and long lucid dream.”  He smiled and looked at me, saying, “This is not a lucid dream.”  And at that moment, I realized that he was correct, since I did not recall becoming ‘aware within a dream’.  So, you will see that sometimes when you are ‘aware’ in an inner dimension, but can not recall how you became aware (because there was no dream), then you may be at another deeper inner dimension.
Now in my books, I suggest one way to explore this:  Become lucid, stabilize the lucid dream, and then announce, “Show me the next level” or “Take me to the next form/dimension.”  Normally, things will suddenly change, and you may find yourself aware in the kitchen of your house (probably in what people call an OBE state) or you may find yourself moving through a vast space of blackness before you suddenly come to a new inner environment.
As I mention in my first book, though, you can have experiences with ‘light’ and with geometric figures of profound energy.  In fact, I came to a certain level of realization, and would fall asleep and the entire night was ‘blue light’.  No symbols, no action, just blue light for the entire night!   This led to an even deeper understanding and some larger experiences (too complicated to explain).  But I bring this up to say that even realms of light seem reachable.
In many ways, I think of the correct path of lucid dreaming as Lucidly Following the Light.  Or perhaps, another person might say the correct path of lucid dreaming as Lucidly Following Creativity.  If you follow creativity deep enough, then surely you will eventually come to the realm of the Creator.

At my workshops, I routinely meet people who had a very active lucid dream life — then it drops off — and they want to get it re-started.
From their experience, I have seen a few common situations, and resolutions:
Got Too Busy — Some people have a life change (marriage, kids, new job, anxiety/depression, etc.) and their time and interest in lucid dreaming plummets.  Years later, life stabilizes and they want the lucid dreams to ‘return’.  😉  For these people, the resolution normally involves generating interest and enthusiasm in lucid dreaming again — and having ‘concrete goals’ instead of mushy ‘I-wanna-get-lucid’ but can’t explain why goals.
Sudden Stop — Some people tell me that their lucid dreaming just stopped.  And they have not had any major life change or anything.  😉  Normally with these people I encourage them to return to the last lucid dream or two — and see if something happened in the lucid dream that may have asked them to grow or to change, or resulted in resistance or fear.  Often when they return to the last remembered lucid dream, they can see some ‘issue’ (and have ego resistance to it, or fear of it).  The resolution?  In the waking state, dialogue with the issue until you resolve the fear or the resistance.  Done thoughtfully, your lucid dreams will return!
Hurt Feelings — At a workshop, a guy told me he had not had a lucid dream in a couple of years.  I ask him, “What changed two years ago?”  It took a couple of days, but then he realized that two years ago, he was lucid dreaming a lot.  He got so excited that he asked his dad to watch the movie, ‘Waking Life’ about lucid dreaming.  At the end of it, his dad said, “What a bunch of rubbish!” and said that lucid dreaming was stupid.  The guy cared about his dad — and suddenly stopped lucid dreaming.  😉  Having this realization, and moving past the hurt, his lucid dreams began again.
Medications — If you look closely, some medications have ‘side effects’ in really small print!  And occasionally you will read, “Inhibits dream recall” or “May create vivid dreams”.  If you notice that your lucid dreaming life changed around the same time you began a medication, then check into the side effects.
Other…. Sometimes people read something silly or adopt a new belief where ‘lucid dreams’ appear frightful — and their subconscious fear makes it difficult to lucid dream.  Research has shown that lucid dream is natural, and happens spontaneously in children as young as 6 years old (and a bit more than half of the population).  Understanding that lucid dreaming seems a natural thing should dissipate any concerns.  😉 Dreams happen!

It sounds like you have been very observant about your dream and lucid dream life!  Wonderful.
Regarding your lucid dreams (3 & 4), all of us have some percentage of #3 ‘lucid play’ and #4 learning-lucid-dreams.  As we develop as lucid dreamers, I encourage everyone to see the potential in using lucid dreams to Access Creativity, Engage the Larger Awareness, Promote Mental/Emotional/Physical Healing and Personal Growth.
The transformative potential of lucid dream exists, but we have to work with it and intend our way to it.  😉
Regarding #5, yes, there are experiences which do not seem like dream-dreams.  Instead they seem like viewing the world through the eyes of another.  I began to have these as a teenager on rare occasion.  In fact the first time, I found myself sitting in the living room of an older man wearing a wool coat in a simple room with ‘gas’ lights, who had a glass of red wine in his hand.  For me, as perhaps a 14 year old kid, I had never drank wine because my family was against alcohol consumption.  Anyway, the man lifts up his glass and (I sense) the excellent wine explodes in his mouth with a wonderful taste!  I woke up thinking, “Is that real?  Can tasting wine result in that experience?” — and it’s only been a few times in my life when a drink of wine has mimicked this dream experience.
In a chapter that my editor tossed out of my first book, I shared a number of these experiences in a focused chapter on the possible meaning of this type of experience.  I hope to put it in my third book (which I intend to publish some day).
Enjoy your journey of awareness!

Thanks for sharing your lucid dream experience!
In my books, I cover this more deeply — but yes, lucid dreaming can have such a vivid sensory and perceptive experience that it seems ‘little difference’ from waking reality.  😉  I have to imagine that this fact helps you and other lucid dreamers consider the nature of perception and the mind.  And will someday push science to consider ‘what, if anything’ exists out there!
In my talks and workshops, I point out that we spend about 23% of our sleep state in dreaming.  By age 11, we have spent an entire year in the dream state.  By age 33, we have spent three years in the dream state.  Because of this, it does not surprise me that someone would develop ‘dream memory’ (and recall past experiences in dreams and lucid dreams, and ‘knowledge’ or ‘language’ experienced within the dream state).
However, since science and culture largely devalue dreaming and mis-understand lucid dreaming, this too may take another generation to receive adequate attention.  Like you, I see written words in my lucid dreams which I wake and take to Google Translate — only to discover that my inner awareness showed me a word from ancient latin or Greek or Spanish.
In my first book, Lucid Dreaming – Gateway to the Inner Self – I show others how to take lucid dreaming much deeper.  If you haven’t had a chance, I hope you can check it out and read it.
Enormous potential exists in lucid dreaming for real understanding and insight!  Lucid wishes!

Thanks for writing about your experience.  I think you can make a very positive improvement — so I hope you will listen to this advice.
By your description, it does not sound like a ‘lucid dream’.  Instead, it sounds like you are describing the common characteristics of (what people call) an OBE or out of body experience.  In my first book, Lucid Dreaming – Gateway to the Inner Self, I have half of chapter on the similarities and differences between ‘lucid dreams’ and ‘OBE’s’ — and your experience sounds like an OBE.
Why do I say that? Here’s why:  1) In lucid dreams, you normally do not “see yourself” laying in bed — however, in an OBE, that is very common, 2) In a lucid dream, you normally do not have “ringing” in the ears — but at the beginning of many OBE’s, that is very common, 3) In a lucid dream, normally a person is very happy because they realize, “Hey this is a dream!” — but for people who are having OBE’s, they often feel frightened or terrified, since they do not understand what is happening.
So, here is my advice:
Calm down.  Try to remember that this event has happened before — and you have always been fine.  So calm down, and take it easy.  Also, think to yourself, “Millions of people have had this experience too. So it is not that big a deal.”
Most OBE experts consider the ringing in the ears, as a reflection of the shift in ‘energies’ or ‘frequencies’.  So again, it does not harm a person.  It is noisy — but at a certain point it will stop.
Read books on how people use the OBE state.  When you do so, you will see that people use it to have amazing adventures!
As you read the books, you will see how to ‘end’ the state too.  Knowledge is power.  So read up on it, and learn about it.
Finally, understand that there are a lot of people who have experienced this state, and the ringing in the ears, and seeing themselves on the bed sleeping — And they figured it out, because they read up on it, and in the process, they lost their fear of it.

In my interviews, I often tell the story of lucidly asking a dream figure, “What do you represent?” — and unexpectedly hearing a non-visible ‘voice’ give a partial reply.  I then asked a follow up, and received a full reply to the question.
But in the morning, I wondered, “Why did a non-visible voice respond?  Does this mean that there is an awareness behind the dream, which lucid dreamers can communicate?”
In my first book (which I encourage you to read), you can see how I began to look into your fundamental question — am I just hearing an ‘echo’ of my own thinking?  Or does this responsive awareness represent a different or larger aspect of the ‘self’ of which Robert W is a part?
Now in your experience, you say you received a response (and by the sounds of it, the response seemed to make sense) — but on waking, you wonder if this experience just constitutes ‘expectation’, right?  In general terms (and as you would read in my book), if you continue along these lines of investigation, you will encounter “unexpected” responses from this non-visible voice — as many lucid dreamers have.
In my first book, I share many examples from lucid dreamers around the world — who shared their experiences — and ‘heard’ the non-visible voice disagree with their request and offer an alternative explanation.  For example, the leader of LD4all.com, PasQuale, asked, “Show me the beginning and end of the universe” — and then the non-visible voice responded, “The universe has no beginning and has no end.  The universe is an everlasting cycle.”
And I can provide many more examples where the non-visible voice offers an alternative explanation, or unexpectedly cautions the lucid dreamer to “not” proceed with their request (and explains why).
As you will read, you see this non-visible voice respond with these characteristics — perception, apperception, memory, reflection, judgment, affectivity, creativity, etc. “all in subliminal form”.  I bring this up, because Carl Jung said that if we have an inner ego or inner self, it should have all of these characteristics!  Lucid dreaming, as you go deeper, and interact with this non-visible awareness — it shows all of these characteristics.
Anyway – great question — and please keep exploring.  You will naturally begin to see that something more than placebo or expectation is happening here.

Glad that you have enjoyed reading my first book.  Congratulations on your lucid dreams!
Regarding your question, many lucid dreamers have reported that the ‘dream’ seemed created in order to help them become lucid.  For example, there might be an advertisement on a billboard, asking, “Are you dreaming?”, and they see it as they drive by.  Or the person might meet a dream figure, like your’s, who asks them questions about their awareness and tries to push them to become lucid.
Why does it happen?  Why don’t they just say, “Hey Luciana, this is a dream!  Look, I’m floating. This IS a dream!”??
When you read thousands of lucid dream reports, you see something like an unspoken ‘prime directive’ where dream figures can hint at and suggest ‘This is a dream’ — much like a elementary school teacher can respond to your question by hinting at the answer, but can not tell you the answer since the teacher wants you to learn how to come to the proper conclusion!
Having said that, I have read some dreams where the person argues with the dream figure, and assures the DF that this can not be a dream, since it is “real”.  So some dream figures finally get to the point of directly stating, ‘This is a dream’.  A person might state that those DF’s symbolically represent or act as the person’s desire to become lucidly aware.
In my books, I note that “All dream figures are not created equal” and that they vary considerably.  It’s great to hear that you have one who seeks to help you become lucid, and gain from the experience.  😉

Thanks for sharing your experience with lucid dreaming and false awakenings.
When it comes to false awakenings, it helps to look at the “process” by which they occur: 1) Normally, something happens and we ‘expect’ that the lucid dream is ending and we will wake, 2) The energy of that ‘expectation’ (of waking) serves to ‘create’ a waking-type experience or false awakening, 3) And then, either we wake to physical reality or we get caught in a bit of a loop, and keep ‘expecting’ and ‘creating’ and expecting and creating ‘false’ awakenings.
For myself, I think false awakenings happen at a certain stage of the lucid dreaming’s life (that period when they know enough to learn when a lucid dream seems likely to end, and therefore they ‘expect’ to wake).  However, since you are in the midst of it, I think you may want to do a ‘reality check’ (e.g., pull your finger, try to float) whenever you feel like you have ‘wakened’ from a lucid dream.
Altogether, I believe that you will eventually move through this phase (and then false awakenings will be exceedingly rare).
To your next point, that sometimes the waking state gets very dreamy or dream-like.  For someone who is going deeper on the lucid dreaming path, this happens. I encourage everyone to do these things:  1) Get outside and take a walk, sit in the sun, get involved in something (even if it is throwing a frisbee), 2) Do not sit in your home all day listening to music, or meditating, 3) Get together with other people — call up your friends and meet them for pizza.
Finally, in my second book’s final chapter, I mention a waking technique whereby you can see how this perceived experience of waking gets ‘projected’ outward into the waking world.  By doing this waking technique, you will get a glimpse into a more basic understanding of the creativity of perception.  This may help you develop realizations for a stronger foundation, along the path.

In my books, I mention that in WILDs (wake initiated lucid dreams) where the person seeks to fall asleep while maintaining their awareness — and then become lucid in the dream scene that forms — it can seem like “moving into” the dream through a portal or doorway or bubble.  Many others, like yourself, have also noticed this.
If you have a problem with the ‘frame’ or the doorway or the portal collapsing, then I think it may have to do with the approach.
First, a newly formed dream seems quite delicate, but if you focus within the action in it, then that automatically helps you ‘get there’.  By contrast, to focus on the gate, the door, the ‘frame’ or the portal, makes the dividing line more powerful as a barrier to entry.  So focus on ‘where you want to be’ — and as you focus on what is ‘inside’ the frame or the portal, then your awareness moves there naturally.
However, if you have already focused on the frame or gateway or portal, then you will need to use it.  If you can, ‘grab it’ and pull yourself through it and “into the lucid dream”.  It may literally feel like pulling yourself into something.
Finally, do not get too excited or too emotional, since that can cause the lucid dream to collapse.
--
To practice this in the waking state, stand outside of an open door (maybe 5 feet from it) — and then focus on the space inside the ‘room’ (meaning, ‘focus’ pass the doorway and into sitting on the chair in the room).  Cast your awareness into the room.  Do not focus on the door or the division between you and where you want to be — instead, see yourself already in that space, sitting in the chair, observing the room from the chair perspective.

Thanks for sharing this interesting (sub-lucid) dream — and congratulations on your soon to be born son!
In many cultures (e.g., Korean), “naming dreams” occur before the birth of a child.  In fact the parents will pay attention to their dreams, expecting to receive a “name” for their child!  Of course, many cultures naturally accept that their child may be reincarnated, and have a history and lives before their birth into this new one.
I recall hearing of an expectant couple in Minneapolis, who heard in a dream (on the same night!) that their child wished to be called a specific name.  Like you, they had not really considered this name — but they both felt amazed to have such a dream on the same night.
If I had this experience, I would do a few things: 1) I’d tell my partner about this dream, 2) Then I’d ask our families, “In our family tree, is there anyone named Reinhardt?”  You may discover that a great grandfather actually has that name on the mother’s side.  And finally 3) I’d discuss with the mother about a middle name being Reinhardt.
Then after the child was born, I would let it grow up and see if it naturally has an affinity for the name  — or for a country or culture where that name is more popular — or has physical connections to a country (perhaps, it has blue eyes, even though everyone else in the family has brown eyes).  For example, as a child, if it hears traditional German music and naturally begins to do a folk dance, then I would think, “Hmmm, there may be something to this Reinhardt business after all.”  Or if the child goes to the library for the first time and finds a travel book on Germany and looks spellbound by the images, then there may be something.
A professor at the Univ of Virginia, Dr. Ian Stevenson, did some fascinating research on children, who spontaneously began to talk about past lives.  Sometimes they grew up and ‘recalled’ their ‘other home’ or their ‘other family’.  In some cases, the information actually connected to another family sixty miles away — or the evidence could be validated in historical records.  Stevenson has a number of books and YouTube videos about his research.

I provide an answer in my second book, Lucid Dreaming Plain and Simple, co-authored with Caroline McCready.
In the dream state, we normally have REM — rapid eye movement.  Our eyes literally move.
But in lucid dreaming, when we decide to stare fixedly at something for more than a few seconds, this becomes un-natural or abnormal to the REM state — and then that tension (of doing something in defiance of rapid eye movement) causes the lucid dream to collapse.  When it collapses, the lucid dream ends and the matter gets resolved.
While you may seem able to look at something in a non-lucid dream for a long time, I do not believe it is truly possible (because you lack the lucidity or conscious awareness to ‘stare’ fixedly).  For example, in a nightmare, you may be looking at the witch — and then without realizing it, you look at her pointed nose, and her pointed shoes, and then the wart on her face.  When you wake from the dream, you think “Oh I really looked at the witch for a long time” — the actual experience is that you looked at differing aspects of the witch (so you have slight eye movements, focusing on various aspects).

In my book, Lucid Dreaming Plain and Simple, co-authored with Caroline McCready, I noted that vitamin B 12 and B 6 may help with dream recall, especially taken before sleep.  So you may find that helpful.
Some people have incubated dreams, where they suggest to their unconscious that “Tonight I will have a dream which will clearly show me foods or drinks that will increase my number of lucid dreams.”  Then during the night, they take extra notice (and notes) of what foods or drinks appeared in their regular dreams.   You can also request (on a different night) “Tonight show me foods or drinks that I should avoid to increase my number of lucid dreams.”
Ed Kellogg has written about this for the magazine, Lucid Dreaming Experience, at www.DreamingLucid.com  Go to “Index” and scroll to Ed Kellogg’s name, then look for The Dream-atarian Diet.  He would regularly see foods in either a positive or negative light in his dreams and/or lucid dreams — and when seen in a positive light, then he would add or increase those foods — but if seen in a negative light (e.g., the food falls into the dirt), then he would decrease or eliminate it from his diet.
Regarding your observations, many lucid dreamers have noted that being outside in the sunshine can have an influence on dreaming and lucid dreaming — or simple hard work (going to sleep exhausted) can lead to spontaneous lucid dreams.  Also lucid dreamers notice that feeling content and at peace, while being more mindful during the day, often leads to an uptick in lucid dreams.

Congrats on your success with lucid dreaming!  Even though things seem difficult or different now, you will likely find that you get back on ‘the path’ at some point.
At workshops, I often mention why many people begin having difficulty with lucid dreaming (who had previously been wonderful lucid dreamers):
 If they go back to their last ‘significant’ lucid dream (before the difficulties began), they often discover a lucid dream which challenged them, made them fearful or made the ego want to block lucid dreaming (as too powerful or knowledgeable),
 If they go back to their last ‘significant’ lucid dream (before the difficulties began), they often discover that their life has changed — they fell into a funk or depression — they lost motivation — or they had some significant life event (a break-up, a divorce, got a 80 hour a week job, or something)
 If they go back to their last ‘significant’ lucid dream (before the difficulties began), they often discover that they have begun to take some medication prescribed by a doctor (or they have started to self meditate by smoking/drinking).
Now sometimes people take a number of months or years to “Work out” the issue (for example, the ego may forgive the larger awareness, see how the ego connects with the larger awareness and now allows incredible lucid dreams again).  Or the person may work out the “fear” that was literally or symbolically shown in the last significant lucid dream.  In other words, the person made the inner change or inner resolution of the issue — and their regular lucid dreaming returns.
Does any of this make sense?
If you sit down and think about it, is there some ‘fear’ (which may even be philosophical ) that is keeping you literally and symbolically “partially paralyzed”?

When you look at a large group of experienced lucid dreamers who have expressed a question to the ‘awareness behind the dream’, you see a vast array of results.
Some people report having a fascinating conversation.  Some report getting a very clear and meaningful response (verbally or visually, as in the lucid dream changes or objects appear).  Some report Zen like or cryptic responses.  Some report very short responses.  Some gibberish.  Some have the interesting case of having the question get caught in their throat — such that they find themselves unable to speak in the lucid dream!
And sometimes, the questions or requests asked seem off-base (so it’s kind of a Garbage In Garbage Out phenomena).
As I write in my books, lucid dreaming is ‘dynamic’ — since we can decide on actions and intend for things to occur — however, our beliefs, expectations, focus, intent/will and X (the larger awareness) help to co-create the response.
Normally, we see that the action in the lucid dream reflects our Beliefs and Expectations (along with our conscious intent, too).
For example, if I believe that I can easily fly through the wall, then I can! I believe it is easy, and it becomes easy.
But if I fly up to the wall, and then think “Oh, this will be difficult — the wall looks very solid” — then suddenly, I bounce off the wall. I can not fly through it. My belief and expectation of ‘difficulty’ gets reflected in my experience.
In lucid dreams, we do not escape our beliefs, feelings and issues.  You write that ” I feel this is related to my strong hesitance to verbally discuss dream lucidity in person with others (though I can discuss it through writing). A (public) speaking fear is not something I have ever battled in my life – except with this topic, despite my intense interest in it.”
Reading this, it suggests that you have some ‘belief’ issue about hearing from a non-visible awareness.  That ‘belief’ keeps you from having a more direct and lengthy conversational interaction — just as my belief in the difficulty to fly through the dream wall makes it impossible.
When you ‘change’ your belief, when you ‘change’ your mind, then this will be much easier.
So tell me, why is it hard to discuss this with another?  Do you have a belief that it seems crazy?  That you will look too new age spiritual?  When you resolve the issue, then you will allow the larger awareness its fuller conversational ability.

Thanks for sharing your experience.
As a fundamental rule, we learn that ‘dreaming’ reflects the mind (and its beliefs, expectations, focus, intent/will, etc.).  In lucid dreaming, we have the ability to see ‘how’ this process of reflection occurs.
In this particular case, there are a few things that may be behind the issue of the dream collapsing:
You have come to believe that creating an energy ball and saying “Chi” will transport you from regular dreaming into the lucid dream state.  This is your belief.  The problem with this approach is that it may “reflect” or help to create too much ‘energy’ in the mental atmosphere, which makes the lucid dream more unstable.  (For others reading this, if you have too much energy in a lucid dream, then it often becomes unstable and collapses.)
Therefore, you may want to ‘move into’ the lucid dream by simply pulling yourself into it, stepping into it, or something visually appropriate to the lucid dream situation (and not perform the ‘chi’ approach).  If that results in a stable lucid dream, then you have resolved the problem.
The next issue may be the general approach, since you lay in bed for a couple of hours, relax, and then see hypnagogic imagery and then “cross the passage” into dream awareness — basically you are doing a Wake Initiated Lucid Dream approach to lucid dreaming (meaning that you are moving from the waking state into a lucid dream).  I would suggest that you not spend two hours of relaxation (after a half hour of meditation) — it’s far too long.  I would suggest that immediately after meditating, you then tell yourself that your next dream will be lucid, and fall asleep.  I say this because your ‘mind’ will be much more in the sleep/dream state naturally (after 30 minutes of meditation), and you will move into a deeper sleep state, where your lucid dream will be more stable.

Congrats on your lucid dreams!  Glad to provide some feedback.
For others reading this, I have suggested lucid dreamers can ask to have a ‘conceptual experience’.  For example, a person interested in enhancing their spiritual understanding could ignore the dream figures and ask the unconscious mind or larger awareness in the lucid dream, ‘Hey, let me experience unconditional love!’ — and then normally the lucid dream changes and they begin to have the ‘experience’.  Some people have reported profound experiences, saying that they never understood the true depths of unconditional love, before they asked for this in a lucid dream.
As I mention in my books, lucid dreaming shows us that this ‘state’ has rules and principles, which we need to understand in order to explore and experiment successfully.  One such rule involves the importance of wording.  I want to point out that if you asked, ““Hey dream, let me feel unconditional love for 1 minute!” — then this may result in a different experience, than this intent, “Hey dream, let me experience unconditional love for 1 minute!”
Just the simple change in the request (asking ‘to feel’ versus asking ‘to experience’) may be enough to alter the result significantly.  So, everyone should always look at the wording of their request, and tweak it (in future experiments) to see what, if anything, changes.
The second point about your experience, I have to wonder about the situation at the time of the request, since you state, “I become lucid in a dream. I pretty quickly look to the sky and yell, “Hey dream, let me feel unconditional love for 1 minute!”  I have noticed that some people become lucid, and within seconds launch into their goal (whatever it might be).  Some do this because they ‘believe’ that the lucid dream will end quickly — some do this because of the excess energy in trying to achieve the goal (which suggests an element of fear in not achieving it, or neediness or grasping) — whatever the case, a person has to look at the resulting experience in light of their own ‘mindset’ at the time.
For this reason, I encourage people to become lucid, then stabilize the lucid dream — which means you create a good foundation for your exploration, you settle your mind and clearly recall your intent, and experiment thoughtfully and mindfully (not suddenly, like someone grasping for something).
This idea of a ‘mindset’ in lucid dreaming also connects with my chapter on the ‘Reality Creating principles in lucid dreaming’ — which I see as Beliefs, Expectations/Emotions, Focus, Intent/Will and the X or larger awareness/unconscious.  Since lucid dreaming is mentally reflective, it reflects back to you, your expectations.  If you ‘expect’ disappointment, then it reflects that back (normally). So a lucid dream experience has to be understood in light of the person’s ‘mindset’ at the time of the experience.
Finally, since you mentioned that it “fizzles” out soon, I would also remove the “one minute” clause to your request (i.e., I pretty quickly look to the sky and yell, “Hey dream, let me feel unconditional love for 1 minute!”).  If you want the ‘depth’ of the experience, I would not put a time limitation on it.  So the others reading this understand, I have mentioned in my talks and books, that when I decide to explore something utterly profound, I often request that I have the experience for “one minute” — and then normally the lucid dream profoundly changes, I feel the depth of the experience, and at one minute – boom – I’m back in a regular lucid dream.
Again, congrats on your lucid dream explorations!  By considering these points, you may find the one thing that needs changing to allow you to take this deeper, and experience, “unconditional love”.

Lucid dreaming provides a path to ‘meet’ your self/Self — and along the way, we meet unexpected and limiting beliefs, expectations, and ideas — as well as ‘events’ that the ego mind interprets as ‘fearful’ (even though they come in “the way of health and wholeness”) — and more.
By your initial inability to ‘speak’, I think this symbolically reflects the approach-avoidance conflict about the larger awareness or the idea of surrendering (and suggests that you have some ‘fear’ in this area).
Even though “the face of the demon” (as you interpret it), “was very much like mine” (or your own face), this might have given you a clue that ‘you’ are dealing with ‘you’ — but the fear distorts the experience!  So yes, I would say that there seems some belief, which you have not resolved.  Finally, as experienced lucid dreamers learn: if you run away, then was has been resolved? Nothing gets resolved through the medium of fear or avoidance.
Lucid dreaming helps us learn, and helps us to develop fearlessness and trust, as we discover the actual nature of things.  But for some, they may come ‘face to face’ with unarticulated beliefs, and learn that they must ‘first’ discover their beliefs, and resolve them –before they go much deeper on the lucid dreaming path.
Yes, to your final point, it sounds like it began in the Void.

Dream-masks, which purport to help a person become consciously aware in the dream state, have been around since the 1990’s.  The first popular one, the NovaDreamer, emerged from a group of people connected with Stephen LaBerge.  Basically, it noticed rapid eye movement, and then flashed red lights — which hopefully would become incorporated into the person’s dream and serve as a cue or prompt to become lucid.
Since that time, various other masks have been made.  Some flash lights at random times (when you may or may not be in REM).  Some also may make audio cues, (“Jim, this is a dream – you are dreaming”), some vibrate, and so on.
I recall when my first book came out.  A lucid dreaming fan sent me his home made lucid dreaming mask — which actually looked very professional.  I set it up, put it on my face and went to sleep that night.  But in the middle of the night, I realized that it was not working!
So the next night, I took more time, and set up the mask.  That night, I found myself driving down a snowy road, and a car went into the ditch — suddenly it began to flash its red lights…. Red lights!?  Suddenly it occurred to me, ‘Red lights — it must be the dream mask!’ and I floated out of the car, lucidly aware — and went on to have a wonderful lucid dream.
But when I woke in the morning — I realized the dream mask did not work at all.  Instead, my expectation to see red lights flashing happened — and I concocted a dream in line with my expectation.
I don’t want to endorse any dream masks at this time.  There may be some that offer a person real value — but caveat emptor, and best wishes as you search for one.

Lucid dreaming is a profound path, and it appears that you may have a natural talent for it.  The important thing: Realize that your larger awareness will make you aware of things, which seem unusual and energetically strange (such as your experience “floating in a kind of red mist, full of symbols and archetypes. I got scared. My dream seemed to collapse”).  If you can simply accept these experiences and observe them with a feeling of trust, then you will learn more easily and rapidly.
Because of its profound depth, lucid dreaming requires a sense of trust and fearlessness — and in my book, I discuss the Reality Creating Principles of Belief, Expectation, Focus, Intent/Will and X – the larger awareness.  In your lucid dream, you had the capacity to manipulate the ‘dream’ by virtue of these Reality Creating Principles.  However, when you reach out to the larger awareness and request “More awareness!”, then you have to accept what comes (e.g. the red mist full of symbols).  😉
Through such experiences, we learn the difference between the ego (which still fears for itself) and its larger awareness (of which it is part, but fails to accept this fact).  Lucid dreaming, naturally, leads to an integration of ego and larger awareness — or towards a more ‘spacious mind’.
To your questions:
So after the lucid dream “seemed to collapse”, you entered a “false awakening.”  Normally, because of our thinking in the false awakening, we fail to realize our situation (and lack lucidity).  But sometimes, a person can think, “Wait a second – how did I get here?!” — and return to full lucidity (that is, they use the false awakening as a springboard to become lucid again).  False awakenings normally occur because “we expect” to wake — and the energy of that “expectation” creates a new scene.
Why does the emotional intensity of the induced lucid dream seem less than your two spontaneous lucid dreams?  I can only guess — but I have seen instances where the person’s larger awareness seeks to place them on the lucid dreaming path — and therefore to engage the person and their ego, the larger awareness creates a profound lucid dream of depth, profundity and emotional intensity!  The larger awareness creates the lucid dream’s circumstance and direction (based on the inner need of the person) — and the lucid dreamer feels ‘wowed’ by it.
However, when the lucid dreamer begins to have their first induced lucid dreams, then he or she ‘begins’ their conscious education in the Reality Creating Principles, and now has to deal with their own beliefs, expectations, focus, and even ‘fears’ (when they encounter strange experiences).  This education represents the foundation for their future growth — so that in the future, they can ‘create’ lucid dreams of profound depth and emotional intensity on their own accord!  But you (the waking self and ego) must learn.
It is not enough that your larger awareness knows.  You must know.  You must learn.  And when our larger awareness sees us request ‘more awareness’ — and then fear the resulting experience — then it shows our ego self that we have much to learn.  But that is the nature of growth!

Thanks for your question about lucid dreaming and interacting with one’s subconscious mind or ‘larger awareness’ or ‘awareness behind the dream.’
To those reading this, who might want to try it, I suggest that when you become lucid, and stabilize the dream, that you simply ignore the dream figures and objects, then look up, and ask (something like), ‘Hey dream!  Show me something important for me to know!’  Often, something will then appear, or some entirely new setting/situation will appear.  In some cases, this can be quite dramatic and personally significant (for more info and examples — see my books).
Many people have success with this.  However, some people do not — and often when they send me their lucid dream, it is easy to see that they have a fear or reticence about engaging another level of awareness (for example, they may not be able to speak, or start to feel queasy).
Also, there are some people who do not succeed, but when they share with me their question or request (asked in the lucid dream), it seems almost non-sensical, or over-layed with stuff that is in the ego’s domain.
So my suggestion to you is to try this:
1) Ask simple questions/requests first, such as, ‘Hey dream, show me something important!’ or ‘Hey dream, show me something hilarious!’
2) Then positively expect a response — and if you get a response, examine it (since sometimes the subconscious speaks in symbolic terms, which the ego may not understand) — or you may get a vocal response.
3) Also examine your self — and your beliefs and emotions — if you do not ‘expect’ this to work, then you yourself may inhibit a relational response — or if you ‘believe’ that you control the entirety of the lucid dream, then you have already set up a lucid dream reality where this idea is not accepted.

My technique of looking at your hands before sleep, while quietly repeating in your mind, “Tonight in my dreams, I will see my hands and realize I am dreaming. Tonight in my dreams, I will see my hands and realize I am dreaming….”  — ideally, this should psychologically ‘condition’ you — and when you see your hands, you naturally respond, “Oh, this is a dream!”  😉  However, it sounds like you had dreams of doing things with your hands, but you failed to think, ‘Wait this is a dream!’
So the good thing is that you got half way there — you saw your hands!  But the unfortunate thing, the second piece (realizing, “Oh, my hands — this is a dream!”) did not get connected to the first part.  😉  So, a person could do this:  Try to connect the two parts more firmly!
You might do this by looking at the palm of your hands for 20 seconds (about ten times during the day) and thinking or imagining, “Oh my hands, this is a dream!”  You really have to pair the sight of your hands with the immediate thought, “Oh, this is a dream!”
Now there may be another issue…. which I hesitate to bring up.  You write that you have kept a dream journal for 40 years, lead dream groups, etc.  It may be that you have an inner belief opposed to lucid dreaming (I say this because lots of ‘long time’ dream workers told me that lucid dreaming was basically evil because you controlled the dream — and in my books, I spend a lot of time showing how it does not mean ‘control’ of dreams — but instead means ‘more aware relating’ to the dream, since you more consciously relate to the contents of the dream).
Can I ask — did you, when you first heard about lucid dreaming 20 or 30 years ago, think ‘Oh, how horrible to control your dream!’ (or something somewhat similar)?
For some people, they have to ‘resolve’ their philosophical issue with lucid dreaming — before they will ‘allow’ themselves a lucid dream!
For some people, their ego/waking self wants to have a lucid dream, but they feel on deeper levels that it is somehow wrong, or not natural.  This situation leads to an impasse’.

Thanks for submitting your lucid dream, and your excellent questions! Because lucid dreaming seems mentally reflective (it reflects our beliefs, expectations, focus, etc.) and also mentally dynamic (when we ‘change’ our beliefs, then the lucid dream changes), I can more easily analyze a lucid dream when someone provides details of what they were doing and also ‘thinking’ while in the lucid dream.  So thank you.
Relative to your question (about not receiving a response, even though you felt confident), I want to look at precisely what happened:  “Remembering the advice from your lucid dreaming book, I confidently call out to the consciousness behind the dream, “I demand that all thought-forms disappear and that you show me something meaningful.” However, none of the professors disappear and nothing happens, despite the fact that I have successfully used this technique in the past. I feel deflated.”

Okay.
It seems to me that you asked two vastly different ‘intents’ (e.g., 1) all thought forms must disappear, and 2) show me something meaningful) — which I mention in my second book (and probably the first too!) normally results in a lack of response or hesitation or something.  As lucid dreamers discover, the ‘wording’ matters.  Asking to ‘look for art’ that I can create results in a much different experience than asking to ‘look at art’ that I can create (and suddenly it appears — even though the wording differs by one word — for vs. at).  In your case, you ‘ask’ for two very different things — and therefore the energy and meaning of the intent is divided, and in a sense fractured.  When lucid dreamers do this, it normally results in no response or something a bit fractured.

Now it may be that your ‘intent’ of “show me something meaningful” ultimately appears in the very auspicious ending to your lucid dream…”But, just as the professor is turning the knob, some color starts to spread from the center of the large, glass pane. It spreads of its own accord, eventually filling in a relief of the professor and me in the center, surrounded by a circle of the other professors intermingled with colorful butterflies. The dream ends.”

If this was my dream, then I would find it interesting that this happens as the professor turns the knob (since it symbolizes to me that ‘opening’ to this new space of grad school results in the ‘heart’ image that you drew — becoming an image of the professor/grad school surrounded by other professors/education and colorful butterflies — which suggest the metamorphosis of butterflies and lots of positive growth for you as a grad school person leading to a profession).  In a sense, when you ‘turn the knob’ (to grad school), then it will lead to growth, and ultimately personal transformation.
To me, that sounds “meaningful.”

Thanks for your question 😉
In lucid dreams, we ‘see’ that the dream state is mentally reflective and also mentally dynamic.
By mentally reflective, I mean that in the lucid dream, we often see our beliefs, emotions, intents and expectations ‘reflected’ in the experience.  For example, if I ‘expect’ to have a hard time flying through a wall, then I may hit the wall and bounce off — even though it is a dream wall!  Or if I am lucid and ‘fear’ that the river will get bigger, then suddenly the river seems much bigger.  The lucid dream reflects our ‘thinking in that moment.’

So to your situation, I sometimes hear from people who have noticed the same thing:  they wish to ask a question of the non-visible awareness behind the dream (as I talk about it my books) — and suddenly, they can not speak!  They lose their voice.
Because of the mentally reflective nature of lucid dreaming, a person needs to look at their beliefs, emotions, intents and expectations.  For example, do you have a concern about asking a question because you ‘fear’ the response?  If so, it may become very hard to ask a question.
Or do you have some ‘belief’ about the non-visible awareness behind the dream, which makes it hard to ask a question (e.g., some people believe in the ego self, but have a very difficult time believing in something ‘beyond’ the ego)?
Or does the ego self feel conflicted about asking a question — that is, one part of you wants to ask the question, but another part of you resists?  That is the classical psychological situation called ‘approach-avoidance’ conflict (which Wikipedia states is this: “Approach–avoidance conflicts occur when there is one goal or event that has both positive and negative effects or characteristics that make the goal appealing and unappealing simultaneously. For example, marriage is a momentous decision that has both positive and negative aspects.” )
By resolving the belief or emotion or intent or expectation in the waking state, you then have a much greater chance of resolving it, and then easily being able to ask a question of the larger awareness.
Now for beginners to this aspect of lucid dream, I suggest that they start out with something simple:  For example, ask the larger awareness, ‘Hey dream, show me something funny to see!’ or ‘Hey dream, show me something important for me to see!’.  These ‘open’ questions are simple and normally do not bring up concerns or fears about the response.
Please realize that the response may be symbolic or it may be quite literal.  Also, when we ask to ‘see’ then we normally receive a visual response — and if we ask to ‘hear’ then we may get a vocal response.  The exact wording of a question or a request is very important.
Finally, you may wish to look at the dream or lucid dream, where you felt like you were ‘asked’ to go live in your mom’s city.  If you wish to send it to me, exactly as you remember, then I will take a look at it.  Sometimes people are ‘literal’ about things that are actually ‘symbolic’.

Glad to hear of your deep interest in lucid dreaming.  Great to hear that you have read my books and listened to interviews, etc.  😉
When it comes to reality checks as a tactic, you may have a bit of an uphill battle because of a psychological factor called ‘habituation’.  Here you have the quick Wikipedia version of this concept: “Habituation is a form of learning in which an organism decreases or ceases its responses to a stimulus after repeated or prolonged presentations. Essentially, the organism learns to stop responding to a stimulus which is no longer biologically relevant.”
In your case, it suggests that the first period of using reality checks, it works!  You respond to the reality checks and become lucid — and even practicing reality checks seems fun and interesting.  But then — a year later, you get back into the reality check game, and it doesn’t work as well.  Habituation.  You simply don’t respond like you use to — and as a result, the old reality checks do not elicit lucidity or never get noticed in the first place.
What to do?
Try another approach!  In my book, Lucid Dreaming Plain and Simple, I offer a variety of ways — including the idea of Developing a Lucid Mindset (an approach which allowed me to have 30 lucid dreams a month).
Hope that helps!  Lucid dreaming is a journey, and like many journeys, we need to adjust as the situation and circumstances change.

There are lots of lucid dream induction methods (besides WBTB), which do not require disrupting sleep.  So I hope you will check out my book Lucid Dreaming Plain and Simple (co-authored with Londoner, Caroline McCready) for some of these other techniques, and find one that works for you.
When it comes to dream recall, it’s important to ‘play’ with memory, and ‘play’ with dream recall (instead of making it a heavy task, which itself is dis-incentivizing).  During the day, you might want to activate your critical awareness by observing and memorizing things in a playful way.  For example, notice the street names on your way to work, and memorize the list.  Some night you may begin to do this, and realize, ‘Am I Lucid’ seems like a strange name for a street ?!! — and find yourself in a lucid dream (I have had a street ‘sign’ serve as a dream sign – and become lucid — so it’s possible).
You can take vitamin B before sleep and do other things (mentioned in my book) — but my sense is that you have inadvertently made dream recall a chore (and this ‘punishment’ makes it work – so you naturally lessen the work by recalling less).  Who knows — you might want to ‘sketch’ in your dream journal — just sketch it like a play: the Setting, the Issue/Conflict, and the Resolution.  Have fun with it.

Lucid dreaming happens at virtually every age.  A study by Voss, et. al, of 694 German schoolchildren (ages 6 to 19 years) found that 50% had already had a lucid dream — and if I recall correctly, 20% had a lucid dream by age 8.

To your point, some aspects of dreaming may work in favor of the young — since they normally report more vivid dreams, etc.  But I have had some people in their 60’s to 80’s tell me that certain heart medications seem to promote extremely vivid dreams, etc.
The one factor that may never get discussed or dissected by science: personal curiosity, interest and enthusiasm for lucid dreaming (regardless of age).  I have noticed that with sufficient focus, belief and intent, a person of any age can begin the steps to master most any ‘art’ or practice.  So if you can remain disciplined and persevere through the lucid droughts (that even teenagers report), then you should have the capacity to take this deeper.
One thing that I try to promote in my online workshops and books: One ‘quality’ lucid dream will make a greater life difference and personal impact than many dozens of play-around lucid dreams.

I have had sleep paralysis maybe 5 or 6 times.  I learned that if I could focus on moving my finger or toe, then I could break the ‘paralysis’, so that is what I normally did.
Then I learned this technique (to use sleep paralysis to become ‘lucid’) in Ryan Hurd’s excellent book, Sleep Paralysis. He recommends that once you find yourself in sleep paralysis, just relax and ‘imagine’ yourself flying over a nearby lake or park — as you imagine and imagine flying over the lake or park, then suddenly you will realize that you are flying, and having a lucid dream!
In this way, you can use your imagination in sleep paralysis to ‘construct’ a pleasant environment, and then lucidly find yourself there.  Sadly, many people do the exact opposite — they find themselves in sleep paralysis and then imagine someone is in the room, or worse, and these ‘imagined’ events begin to appear around them.  Instead, use the sleep paralysis state to ‘imagine’ flying over a nearby lake, and suddenly you will find yourself in a lucid dream, doing exactly that.
Now, it takes a bit of trust, especially the first time, but lots of people have done this and had fun. So if you can imagine that pleasant scene vividly, then you will find yourself there, lucidly aware. Let me know how it goes, okay? You can have an incredibly joyful experience of lucid freedom.
Stephen La Berge felt that sleep paralysis may occur when the mind awakes, but the body remains in normal sleep (and therefore seems paralyzed).  I feel everyone should try and remain calm, and see the potential of this state to move into a lucid dream

Thanks for your question about deceased dream figures in lucid dreams.  Yes, sometimes they may be merely projections of our own mind.  In some instances, though, they may say something which later we can verify or confirm — and then realize that we received information outside of our own knowing.
Why are you waking up?
I can only suggest a possible reason:
Most people wake when they feel too much emotion.  Therefore, it may be that you feel too much excitement that the deceased dream figure of your grandmother may actually tell you something which is outside of your own knowing (because if true, then this would have significant implications for your view of the world).  So you may have a strange desire to know, and also not to know!  This might explain it.
So your subconscious is not preventing you — it seems your own conscious mind and its emotional response that is either fearful or conflicted about hearing a response.

Glad to hear that you see the value of lucid dreaming as a means of personal insight, exploration and spiritual growth.
In my books, I discuss the importance of ‘focus’ in lucid dreams.  For example, if you become lucid and walk outside, then what do you focus upon?  By observing your mind, you will see that you may be drawn into instinctual energies and focus on the attractive woman walking by, or you may focus on some ‘experiment’ that you scientific mind wished to perform, or you may decide to engage in a spiritual practice, like meditating, from within the lucid dream.  Your ‘focus’ in that moment will determine the course of your lucid dream.
Now, if you become ‘lucid’ and then ‘focus’ on whether or not you can feel your physical body, then guess what?  You will begin to activate that feeling, and may notice that you can now feel the bed sheets or the lump in the bed — while simultaneously being in the lucid dream.  However, if you keep focusing on the physical, then the lucid dream seems more ‘daydream-like’, until eventually you may enter a lucid day-dream type situation.
Because of this, I would suggest ‘not’ focusing on any body sensations at all.  In fact, remind yourself that your body is doing fine and lying in bed, but you need to go deeper into the lucid dream and focus on it.  (Note: Some people may have fears or concerns that make them want to ‘check’ in on their body — but if you can let that go, then it will allow you to use lucid dreaming as a spiritual practice which is your personal goal.)
To go deeper, focus on exploring the lucid dream.
Use Your Focus.

As you note, the energy of enthusiasm often helps a person to have some lucid dreams.  But when the enthusiasm begins to wane or dissipate, then the lucid dreams can seem to become very rare.
I have noted this with many people who come to my personal workshops or on-line workshops.  When they first begin, they feel enthused, ‘expect’ to have lucid dreams and focus on it.  But later, things change.
If you look at it closely, not only does the level of enthusiasm drop, but the anticipation and expectation disappears and the focus shifts to other issues (and away from lucid dreaming).  For this reason, I encourage people to develop habits of ‘playful persistence’ — or create a playful way to discipline themselves to make lucid dreaming a priority — whether reading about lucid dreams before sleep (and you can find plenty of lucid dreams at the free magazine Lucid Dreaming Experience, which I co-edit with a friend, just go to www.DreamingLucid.com ), or doing reality checks during the day, or exploring ‘mindfulness’ practices.
All of these things can encourage lucid dreaming, and keep you ‘in the game’.  Enthusiasm is great, but feelings often fade, so you need some practices and approaches to keep your mind involved in lucid dreaming for the long term!

Yes, for beginners it can seem hard to distinguish sometimes, ‘Am I dreaming?’ or ‘Am I having a vivid imaginary experience?’
In my first lucid dream (about age 10 or 11), I found myself in my public library, looking at books.  So at this point, I assumed that I really was at the public library.  Then I watched a small T. Rex walk through the book aisles!  And I suddenly thought, ‘Wait, dinosaurs are extinct.’  Then I thought, ‘How can I be having this experience?’  Then I thought and concluded, ‘Oh, this must be a dream!’  Then I became aware that I was dreaming.
I bring this up to make clear that 1) Dreams seem real, 2) When we question them or analyze them, we may (or may not) come to a new realization and 3) When we correctly realize, ‘This is a dream!’ then we have achieved lucidity.
So some people spend a lot of time in #2 — questioning their experience!  It looks real. It feels real.  But the context is dreamy.  So you may be getting stuck in the questioning — and failing to come to the conclusion, ‘This is a dream!’
My advice:  Whenever you find yourself wondering, ‘Wow, this seems dreamy….’ — train yourself to immediately perform a Reality Check.  Jump up and see if you float.  Or pull your left index finger and see if it gets longer.  By doing a Reality Check, you can confirm, ‘This is a dream’ (and stop the questioning state….).

Thanks for reading my first book, Lucid Dreaming – Gateway to the Inner Self.  While I have expertise in lucid dreaming, I am not a therapist (and these comments are my own personal viewpoint, which may not be applicable to anyone else’s situation).
In lucid dreaming, we see that ‘expectation’ does play a role in ‘creating’ one’s experience.  If we ‘expect’ trouble from a dream figure, then the figure becomes troublesome.  But if we flip our expectation and expect assistance from a dream figure, then the figure becomes helpful and friendly.
Lucid dreaming teaches us to become aware of our own ‘mind’ and its belief, expectations, etc. — so that we can claim our authority to change our own mind and thereby, change our experience.
Now, this also seems the case in waking life (and lucid dreamers call this Living Lucidly).  In waking life, you can begin, bit by bit, to change your own mind (and therefore, change your own experience).  So you pay attention to your Beliefs, and self talk.  When you hear the worry and concern in your self-talk during the day, you stop and become aware of it.  Then you may question the beliefs — or suggest to yourself a different belief with a small change (that moves you towards your desired goal of being free).
To me, lucid dreaming seems to show that the ‘ego’ self brings these things (fears, worries, anxieties) to the subconscious.  😉  So there seems no need to blame the subconscious for what the ego dumps into it!  The important thing seems to involve the ego becoming more aware and thoughtful about its own thoughts.

Congratulations on your lucid dream!
It seems that you have had a number of ‘pre-lucid’ dreams — for example, seeing your ‘reality check’ in the dream state, but then failing to realize, ‘Oh, this is a dream!’
This may occur as a result of an incomplete suggestion.  During the day, a person may say, “Tonight in my dreams, I will see my watch.”  But fail to add “And realize I am dreaming.”
Then they begin to report dreams of seeing their dream sign (the watch), but never becoming lucid — because the ‘suggestion’ involved simply seeing the dream sign, or seeing the watch.
Lucid dreamers learn that the exact ‘word choice’ seems extraordinarily important.
You have made some progress, but need to value those details (which I try to make clear in both of my books).

It seems every lucid dreamer shares this challenge!
In one of my on-line workshops, a participant told me that she was driving along and saw a truck which had a giant ad for Lucid Dreaming, and I believe even asked, ‘Are you dreaming now?’  Instead of becoming lucid, she got mad, thinking, ‘Look how they have commercialized lucid dreaming!  It’s even being advertised on trucks now!’  😉
In the morning, she felt both amused (that her dreaming awareness had put in that clue) and alarmed (that she missed it, and resorted to getting upset).
In my books, I emphasize that critical awareness needs to become part of our daytime activities, so it will then be more likely in our night-time dreaming — pushing us to lucid awareness.
Notice how you might assume and presume during the day, and rarely question or examine deeply.  In my experience, lucid dreaming increased considerably, when I began to question my waking experience (for example, asking myself when I saw something odd or emotive, ‘Why am I seeing this?’  or ‘How does this event connect with me?’).
By doing that during the day consistently, I began to do it in my dreams, and suddenly would realize, ‘This is too strange — so this must be a dream!’  😉
If you have a chance, please read my books (or re-read them).  Many people have told me that simply doing that increases their lucid dreaming.
Also, observe your thinking and your thought-stream.  Above you wrote “It seems that now matter how strange the situation, my mind will not let me become lucidly aware” — and since the Unconscious is always listening, that kind of thinking is like a hypnotic suggestion (which you do not want!).  Instead, see your ‘mind’ as your ally and your Helper — it brought to you a number of things that should prompt lucidity — so thank it!  See how it helps, and encourage your natural talent to observe and critically examine.

Thanks for sharing your experience of ‘golden light’ through hypnagogia.  As you point out, it was unexpected, did not involve ego dissolution, but became more intense as you allowed it to happen (and had a feeling of bliss).
So, it sounds like to me that you may have stumbled into an introductory experience to what may occur in Yoga Nidra (because something similar happened to me, while doing a Yoga Nidra exercise – though mine also had waves of energy in a golden light environment).
You may want to read up on Yoga Nidra, or try some of their practices.
Lucid wishes!

Okay, you had a pre-lucid dream (so it sounds like you had a vague idea that you might be dreaming) and called two of your friends.
And you wonder, “How valid is the information they gave me in the conversations?”
If this was my experience, I would call the two people, and tell them the dream conversation, and their reply.  Then I would ask them, ‘Does that sound correct, literally or symbolically?’ (since we see in dreams and lucid dreams that sometimes dream figures respond with either literal, symbolic or nonsensical responses).
Here it becomes more complex, since people are people.  For example, I recall asking a friend if she had gotten a new silver convertible car?  She told me, ‘No.’  But later I learned that her husband had purchased a new silver convertible (as I had dreamt) — and she had answered literally by saying she had not (though her husband had).
Then you have the issue of ‘memory’ and subconscious association — as in, ‘Had you somehow heard about the information, then forgotten it, and now dreamt it?’  Or, had you heard some things about these people, then subconsciously associated it together, and dreamt the result of those associations (e.g., a woman mentioning her body temperature in waking life, and then you later dreaming, ‘She’s pregnant!’).
And of course, you have some dream encounters where you get symbolic information, but the waking person, when questioned, does not see the symbolic aspect.  And you have situations where the information seems your projections or simply nonsense.
As you can see, it’s complicated!

The black nothing that you describe sounds like ‘the Void’ (the term that many lucid dreamers use for it).  It normally looks like an endless field of black sparkling emptiness.  Sometimes if you hang out there long enough, a dream or lucid dream will spring up around you.
Your question has some complexities to it (and so I wished I had a bit more information to answer properly) — but here are some points, and suggestions:
Lucid dreaming is a very mentally dynamic state, and mentally reflective state.  If we have feelings of fear, or danger, then suddenly in the space around us, a figure or object appears which may reflect those concerns.  So a person has to observe their mind, and gain some influence over it (if it has a tendency to go to the fear realm).
Sometimes people suddenly report the appearance of a new feared thing in their dream or lucid dream life, and fail to see how it connects to something in their current waking life (e.g., a divorce, a loss of a job, a death in the family, etc.).  It seems a good idea to ask, “Why am I experiencing this now?  What is going on in my waking life which may connect to this?”
Now in a true lucid dream, we see that most dream figures are ‘thought forms’ (that is, projections of our mind).  We can easily determine this by doing things like the following:  If an angry dog is there, then send it peaceful thoughts, love, compassion — and normally it will shrink!  Your ‘mental energy’ directed onto it, causes it to change, because it is a projection of your subconscious mental processes.  Seeing this, you may want to send love, compassion and understanding onto ‘this thing’ that you have encountered, and see what happens.
If you wish, you can also do things, like make protective suggestions before sleep, such as, “Tonight I will respond only to constructive suggestions and influences from my self and others, and refuse to engage non-welcomed  energies.”  Often a simple suggestion like this changes one’s experience.
That would be my advice.  If you have more to say, you can fill out the ‘contact form’ and I will respond privately.

Thanks for your question about lucidly responding to aggressive dream figures. Normally, I tell lucid dreamers that if they can respond to aggressive dream figures by asking a question, like ‘Who are you?’ or ‘What do you represent?’, then they might learn what they are battling. Similarly, if they respond by sending compassion, love and understanding to an aggressive dream figure, then they will normally see the aggressive figure shrink down to something simple or harmless, or maybe even turn into light. By responding in these kind of ways, we often ‘resolve’ things or gain new understandings, new insight.
In your case, it sounds like you went to bed feeling angry, then had this dream, and became lucid. That may have ‘pre-disposed’ you to deciding that killing the dream figure seemed the best response in that moment (so you could release those feelings of anger). Your response reminded you that you have ‘power’ and can stop events from occurring or alter them. For some people, that seems a wonderful lesson (especially if they feel dis-empowered in their life). It’s okay to claim your own power, and use it thoughtfully.
The important point: Understanding the issue (presented symbolically) and lucidly working to resolve it (whether waking or dreaming). In some situations, this may involve a series of steps….. and gradual realizations.

Interesting philosophical question.
On a practical level, one might ask, “What human need is served by two hours of dreaming each night?”  Or we could flip it, and say, “The human needs of dreaming seem so important, that the human organism naturally gives it two hours of focus every day.”
In my books, I note that you can use lucid dreaming for the following:
Access creativity, enhance skils, etc.
Promote emotional health,
Promote physical health,
Explore the nature of dreaming, consciousness, the mind,
and Perform spiritual practices.
So you might consider these as some of the ‘particular needs’ connected to lucid dreaming.

Thanks for your question.  In my first book, Lucid Dreaming – Gateway to the Inner Self, I share how I came across the ‘awareness behind the dream’ in 1985, while doing a lucid dream experiment involving dream figures.  I felt surprised to hear a non-visible voice answer my question!  About 8 years later, in Castaneda’s Art of Dreaming, don Juan tells Carlos that this non-visible voice is called the dreaming emissary.
Now in your specific case, this seems less clear, since you did not apparently ask a question (but did you ask in your mind? such as, ‘What is going on here?’ or something like that?).  If you did not ask a question, then I might consider it a ‘dream symbol’ (like thunder, or a police siren, etc.).
Of course, the simple way to resolve this:  In your next lucid dream, simply ask a question, like ‘Hey, show me something important for me to see!’  When you receive a response, then you will understand that a larger awareness seems accessible in lucid dreams.
The final point — you will interpret the experience, according to your mindset or belief system.  A fearful person will find any evidence of a larger awareness as immediately frightening…. So you have to approach this with an open mind.  I think you will find (like thousands of people who have read my books) that an inner awareness can respond in a lucid dream with clarity, creativity and compassion.

Thanks for your question. I have heard from lucid dreamers around the world, who ask a question or make a request in their own language in the lucid dream. Normally, they receive a response in keeping with their question (which means, if they ask ‘Show me____’ then the response will normally be visual, and if they ask ‘Let me experience _____’ then the response will normally be experiential).
In the Spring 2017 issue of the free online magazine, Lucid Dreaming Experience, at www.DreamingLucid.com — a lucid dreamer, Esther, asks a question of the larger awareness and then is surprised to see certain symbols and a Greek word (which she had to investigate on Google to learn its meaning since she does not speak Greek). I bring this up to say that sometimes lucid dreamers may hear or see words in Latin, Greek, possibly Sanskrit, etc.
So, please use your native language when interacting with the awareness behind the dream. As I point out in my books, 1) the exact wording of the question or request is very important, 2) sometimes it takes a little bit of time to see or hear or experience the response, 3) some questions rarely get responses (and these are ego based ones, such as, “Should I go to college, or should I do work at a bakery?” ), and 4) the lucid dreamer must open up to the experience and ‘trust’ (you can not ask, and at the same time, ‘fear’ the response).

Glad you enjoyed my lucid dreaming books  😉
First, congratulations on becoming lucid!  You recognized that the room was too “nice” and realized that it might be a dream.
When you have that realization that ‘This might be a dream’, then I hope you can do a ‘reality check’ at that moment (and not look for a mirror, etc.).  For example, you can ‘jump’ up into the air and see if you float — if you float, then you know it is a dream.  Or you can grab one of your fingers and pull it, and normally it will get longer, then you know ‘This is a dream.’  Or you can pinch your nose, and if you have no problem with breathing, then you know ‘This is a dream.’
Yes, the dream will seem “real”.  But when you can float or fly, then you will know it is a lucid dream.  For a beginner, I also suggest in my books that they remember these three points:
Reduce emotions — If you get very excited, then the lucid dream will end.  So tell yourself to ‘Calm down’ or look at something neutral, like the floor or your hands, when you see something too exciting,
Enhance your awareness — If you become lucid, and the environment seems very to have low light, or seems dim, then announce, ‘Greater clarity now’ or ‘More awareness now’ and often the lighting will increase.  Also, rub your hands together to activate your sense of being there,
Maintain your awareness — Since everything seems ‘real’, then you need to remind yourself every 15 seconds, ‘This is a dream.’  Or you can decide to sing, and know that you are singing because this is a dream (and not waking reality).  It is important to maintain your awareness of being in a lucid dream.
Remember to be more aware during the day, since this will help you to be more aware in the dream, and realize, ‘Oh, this is a dream!’