A typical reading of the story of Cain & Abel leaves one puzzled. Why does God prefer sheep? What's wrong with a vegetable offering?
I'd like to suggest something else might be going on here.
The name קין comes from the root קנה which means to acquire. The term is usually used when it comes to possessions. Thus, the name קין seems inherently linked to materialism.
הבל, in contrast, can be translated futility. But another meaning of that word would be a fleeting breath, and fellow blogger Steg suggested the word "ephemeral." Things that are ephemeral are short-lived, but more than that, they also seem to connote something intangible. Our lives, to God's view, are ephemeral. We live and in the blink of an eye, we die. Thus, I think a case can be made that הבל, whose name connotes that which is fleeting, is likely to be someone who focuses on spirituality. He would be interested in something beyond the transient and transitory, having reflected on that (and on the meaning of his name).
Now we need to recall an important fact. Prior to the Flood, mankind was forbidden to eat meat. Their diet was comprised of fruits and vegetables. Meat was separate, sacred, something God was permitted and man was not.
So when קין offers his vegetable sacrifice, he is offering God something that he is permitted to eat as well. Here God, he's saying, enjoy the same vegetables I am permitted to enjoy.
In contrast, when הבל offers his sheep, he is offering God something of which he is not permitted to partake. This shows a deep understanding of what it means for something to be קדוש, special or sacred. Something is קדוש when it is separate, a thing apart. To be קדוש is to observe laws of separation. This separation occurs in what Rabbi Joseph B. Soloveitchik refers to as the dignity in defeat, where one holds back one's own power and strength because one recognizes the authority of God over him. One does not sleep with one's wife when she is a niddah. One does not perform work on the Sabbath. One does not eat non-kosher foods. One does not marry outside the faith.
The sin of Adam and Eve was one where they wished to be "like God."
They did not understand that קדושה is about הבדלה, separations. They did not see why one tree had to be reserved for God and was not permitted to them. By eating of the tree that God had set aside, they declared that they did not need to observe separations and lacked an understanding of their role in the world.
הבל, in offering up a sheep, is rectifying their sin. He is saying: God, I understand that meat is sacred. It is only for you to have or enjoy. So I will slaughter this sheep and offer it up to You as a way of demonstrating my understanding that You are God and this is uniquely Yours, not mine. I understand my role and the fact that to be holy is to be separate- and to understand separations and boundaries. By offering this to you, I affirm my comprehension.
This is why God would accept הבל's offering but not קין's. It's also why God warns קין that sin is crouching at the door, waiting for him. If קין does not understand his role vs. God's- if his focus is on inviting God to partake of what he, too, can partake of- then he sees himself as equal to God. קין, like his parents before him, will strive to be "like God" in the sense that he does not accept the separateness of his role vs. God's. And indeed, that is exactly what happens. קין performs the first murder- taking life, which is a right reserved for God. קין determines that he has the right to kill, just as God has the right to end lives. God punishes קין in accordance with his logic. קין saw himself as equal to God, offering God the same vegetables of which he could partake- now the bloodied earth will not produce for him, and he will not grow any vegetables at all. He will be a wanderer and fugitive. The brother who should have been by his side is not there- will not witness Cain's marriage or the birth of his children. Because he has chosen to be a god, he will be hidden from God's face. Cain is condemned to live, and every day of his life will be a crushing reminder of the many ways in which he is not, in fact, a god. What הבל understood originally is what קין will come to understand- there is God and there is man, and the two are not the same.